Repository navigation
feat: add project capacity limits and harden supervision - #219
Merged
Merged
Conversation
kunchenguid#5343) * refactor(bin): share the local Firstmate home walk from the wake library Teardown's walk over the root home and its registered local secondmate homes moves into bin/fm-wake-lib.sh as fm_local_firstmate_state_dirs, next to fm_firstmate_root_home, so a second consumer can count task records across this machine's homes without a copy. Teardown keeps its exact refusal wording through a thin wrapper. * feat(bin): defer spawns beyond a project's declared machine capacity A project whose machine-local resource only serves a few workers at once had no way to tell Firstmate so: every queued item was launched, and the surplus workers spent full-context turns retrying the resource. config/project-capacity in the root home now declares how many workers each named project admits at once on this machine. bin/fm-spawn.sh counts the ship and scout records on the same project origin across the root and its local secondmate homes, skipping ones whose ready PR is recorded, while holding the shared project lock through publication. A spawn with every place held exits 75 before any brief render, endpoint, worktree, record, or backlog move, so the item stays queued; batches report it as deferred. Undeclared projects keep today's uncapped dispatch, and an unreadable declaration refuses rather than guessing the limit. Refs kunchenguid#4237 * no-mistakes(review): Document that capacity matches the clone directory name * no-mistakes(document): Rewrap stale fm-wake-lib root-home doc comment * no-mistakes(review): Dedupe local state dirs by identity to avoid double-counting * no-mistakes(document): Rewrap fm_local_firstmate_state_dirs error doc comment * no-mistakes(ci): I fixed all four Greptile findings. All 14 tests in tests/fm-project-capacity.test.sh pass, and shellcheck at warning level is clean on the changed files. Each new test failed against the old code and passes now. - **ci-1 (spaced names):** a declaration line must give a name its capacity whenever the name is a valid clone directory name. `fm_project_capacity_lookup` now trims each line, skips blank lines and lines whose first non-blank character is `#`, and takes the last field as the capacity. Everything before that field is the name, so it may contain spaces. The old error cases still refuse: a single field is rejected, and trailing text leaves a last field that is not an integer. The library header and docs/configuration.md now say a name starting with `#` cannot be declared. New test `test_spaced_project_name_is_declared` declares `my heavy project 1` next to an indented comment line and gets a deferral. - **ci-2 (unreadable records):** the holder count must never silently leave out a holder. `fm_project_capacity_occupants` now refuses when a local home's state directory exists but cannot be read or listed, or when a `.meta` file cannot be read. The error names the path, and `fm-spawn.sh` shows it in its existing refusal message. New test `test_unreadable_holders_refuse_admission` covers an unreadable record in the root home and an unreadable state directory in a registered local secondmate home, then checks that the spawn is admitted once both are readable. The test is skipped when run as root. - **ci-3 (Orca lock):** any spawn that can become a holder for a capped project must take that project's lock. The lookup now also reports whether the declaration caps any project at all, and an Orca spawn takes the per-origin lock whenever it does. This covers every capped same-origin clone. It also covers some cases where no same-origin clone is capped, because a spawn cannot find clones under other directory names without searching for them. With no declaration file, Orca still skips the lock. The comments in the library and in the `fm-spawn.sh` header are updated. The Orca test now clones the origin as `project-2`, which has no declaration, and checks that its Orca spawn refuses while the lock is held and publishes no record. - **ci-4 (worktrees):** `assert_nothing_created` now also compares the project's `git worktree list` from before and after a deferred spawn. Both tests that call it take that snapshot first. Files changed: bin/fm-project-capacity-lib.sh, bin/fm-spawn.sh, docs/configuration.md, tests/fm-project-capacity.test.sh * fix(bin): declare capacity for a project name that begins with # A clone directory whose name begins with # was skipped as a comment, so that project stayed uncapped. A line is a declaration when the # is written against the rest of the name and the line ends with a capacity; a # followed by whitespace stays a comment. * no-mistakes(document): Rewrap project-capacity library header comment * no-mistakes(ci): Lint 2 fails because this PR's code pushes ShellCheck past its memory cap. ShellCheck ran out of memory analyzing bin/fm-teardown.sh in CI (reason=memory, rc=251, peak about 8.39 GB). On current main the same file passes at about 7.29 GB. **Cause:** the new `fm_local_firstmate_state_dirs` function in bin/fm-wake-lib.sh had a conditional `. fm-secondmate-registry-lib.sh` with a `# shellcheck source=` directive inside the function. ShellCheck followed that source again, inside a function scope, wherever fm-wake-lib.sh is sourced, and bin/fm-teardown.sh is the heaviest root that sources it. Measured locally with `shellcheck --norc --external-sources bin/fm-teardown.sh`: - current main (fd325b1): 7.29 GB - main merged with this PR: 7.86 GB - the same merge without the in-function source: 7.27 GB **Rule this restores:** this change must not make any lint root heavier than it is on main. That function holds the only new nested source in the change. **Fix:** I removed the in-function source, which no caller needs. Both callers already load the registry library at top level before calling the function: - bin/fm-teardown.sh sources it directly. - bin/fm-spawn.sh, the only user of bin/fm-project-capacity-lib.sh, gets it through bin/fm-ff-lib.sh. I also documented the requirement in the function's comment and in the "Requires" note in bin/fm-project-capacity-lib.sh. No behaviour changes. **Verification:** - ShellCheck on head: bin/fm-teardown.sh peaks at 7.12 GB and bin/fm-spawn.sh at 6.68 GB, both with rc=0. bin/fm-wake-lib.sh and bin/fm-project-capacity-lib.sh lint clean. - tests/fm-project-capacity.test.sh, tests/fm-teardown.test.sh (102 ok) and tests/fm-teardown-endpoint-safety.test.sh all pass. Files changed: bin/fm-wake-lib.sh, bin/fm-project-capacity-lib.sh * fix(bin): release the Herdr session lock when reclaim finishes A concurrent resume in another home waits five seconds for that lock. Reclaim is the last presentation change on the recovery path, so holding the lock through the launch tail made the waiter time out. The contributions arm check also freezes its one-second clock, the same way the budget tests do, because an unfrozen clock can tick past before the first forge read. * no-mistakes(review): Keep Herdr session lock through launch handoff after reclaim * no-mistakes(review): Skip the spawning task's own record in capacity count * no-mistakes(review): Restore release test comment above its test * docs: scope PR-ready re-evaluation to a declared project capacity A ready pull request frees a place only when that project declares capacity, so the always-loaded backlog contract should re-evaluate on that handoff only in that case.
… section (kunchenguid#6785) fm-procevent-lavish.sh read labels tag=message rows SESSION-ENDING MESSAGE only when session_ended is true and CAPTAIN MESSAGE otherwise, but the count line always said session_ending_message_count. Several composer messages on a still-open board were therefore counted as session-ending. The count line now follows the same session_ended switch: session_ending_message_count once the session ended, captain_message_count otherwise. Message rows stay out of the annotation count, per triage. Fixes kunchenguid#6743
…henguid#6780) The Herdr presentation lock namespace was the fixed machine-global /tmp/firstmate-herdr-presentation, so on a host where two OS users run Firstmate on Herdr the first account to create it owned it and every teardown from the other account was refused with no way to clear it. Suffix the namespace with the account uid. The owner-uid and mode-700 checks are unchanged, so a foreign-owned or wrong-mode name at this account's path is still refused and never adopted, chowned, or removed. Fixes kunchenguid#4716.
…te (kunchenguid#6809) The OpenCode session plugin's shouldArm kept its own copy of the need test that only looked for in-flight task records, while the turn-end guard decides with fm_supervision_needed in bin/fm-supervision-lib.sh, which also counts registered process-event sources and trusted custom checks. With an empty fleet but any registered source or check, the guard blocked every turn end while the plugin declined to arm - a loop the guard's own repair line could not resolve because it names the plugin as the fix. The plugin now delegates the decision to the shared predicate through bash, keeping the local away-record decline and the x-mode.env arm override. OpenCode plugin test fixtures now carry the real predicate their arming path sources, and the arm suite gains six cases asserting the plugin's decision against the shared verdict over the same synthetic state directories. Co-authored-by: Mia Sun <mia@Bigs-Mac-mini.localdomain>
kunchenguid#6792) * fix(bin): resolve a pending reply only from its own task's status line Remote reply ingestion handed every corr= token in a mate's payload to fm_pending_reply_try_resolve together with that mate's own status log, so one mate echoing another mate's token resolved the other request. Honor a status-file override only when it is the record's own parent_status, and match the corr= token as a whole word. Fixes kunchenguid#6538 * no-mistakes(document): docs: scope remote reply settlement to the asked mate
…nguid#6784) agent-skill-trigger-index claims to be the complete agent-only trigger index but omitted operational-home-layout, session-start-recovery, validation-supervision, ship-landing, scout-completion, and away-quiet-supervision. Add each with its own description's trigger, placed beside the related entries. The decision-hold-lifecycle redirect stub stays out, per triage. Fixes kunchenguid#6503
…nchenguid#6814) * test: share a rename-safe agent stand-in across liveness suites On Ubuntu 26.04, `sleep` is the uutils multicall binary, which refuses to run when invoked through a symlink named after another utility. The Herdr descendant process-walk tests built their agent-named process as a `pi` symlink to the host `sleep`, so the process exited at once, its parent shell was gone before the walk ran, and both cases read `unknown unreadable` and failed on that host. The suite stops at its first failure, so every later case went unrun. The Herdr control smoke test's `claude` symlink has the same construction. The tmux liveness suite already solved this with a host-compiled spinner and a survival-checked `sleep` fallback. That builder moves into tests/lib.sh as fm_agent_standin, and the tmux suite, both Herdr descendant cases, and the Herdr control smoke test now use it. When no stand-in can survive a foreign name, a case skips with the reason instead of failing. tests/fm-test-fixtures.test.sh gains a portable regression with a fake multicall `sleep`, so it bites on hosts whose own `sleep` is single-purpose. * no-mistakes(document): Correct Herdr verification fixture reference * ci: retrigger cancelled shard
…he Stop hook's group is torn down (kunchenguid#6787) * fix(bin): keep the supervision host's pass-through successor out of the hook's process group The successor a main-only pass-through leaves for main shared the Stop hook's process group, so the harness tearing that group down after the exit-2 rewake stopped it. The stop published downtime and the next park's first cycle announced an empty check: rearm-resurface, which woke main again in a loop. Start that successor in a process group of its own, as the hook's own handling successor already is. * no-mistakes(review): Give the at-turn successor left for main its own group * no-mistakes(document): Document own-group successor for turn-start hand-back too * no-mistakes(ci): I made the change you asked for: both new teardown tests in tests/fm-supervision-host.test.sh now call the existing `stop_home_processes "$home"` just before `pass`. The tests are `test_successor_left_at_the_turn_survives_the_hook_process_group_teardown` and `test_pass_through_successor_survives_the_hook_process_group_teardown`. No production code and no other tests changed. The rule broken was that a test must not leave a home's watcher or arm processes running after it passes. These two were the only cases in the changed area that broke it. The other host+hook tests already stop their home, and `test_successor_close_during_main_turn_is_delivered_at_the_next_turn_end` leaves its watcher behind too, but it is an older test you said not to touch. The only reason anything was left over is that the successor's arm now sits in its own process group, outside the hook's teardown. `stop_home_processes` kills the watcher by the pid in its lock file, which stops it no matter which group it is in. **Checks run:** - I ran just these two tests from a scratch copy of the suite (since deleted). Both pass in about 13 seconds. - After each test, a process listing filtered to that test's home directory came back empty once the processes had about a second to exit after TERM. - `bash -n` on the test file passes. - `shellcheck` is not installed here, so I did not lint the file. - I did not run the full serial-2 suite locally. Whether it now finishes under its 30-minute limit will only show on the next CI run
…ters (kunchenguid#6823) * test: use idle composer readiness for Claude tmux guards * no-mistakes(test): Fix attended supervision test expectations and isolate worker state * no-mistakes(document): Correct live guard coverage and readiness documentation * no-mistakes(ci): Captain, fixed SC2100 by quoting the cursor-agent assignment in tests/fm-host-mirror-live-e2e.test.sh. Reproduced the failure before editing; pinned ShellCheck lint on both PR test files, bash syntax checks, and git diff --check now pass * test: preserve attended successor close assertions * no-mistakes(test): Fix attended live test watcher takeover expectations * no-mistakes(document): Correct stale attended guard documentation * Revert "no-mistakes(document): Correct stale attended guard documentation" This reverts commit 8e59d89. * Revert "no-mistakes(test): Fix attended live test watcher takeover expectations" This reverts commit c0b8510.
Merge b062eb9 with upstream ancestry intact. Reserve capacity before house releases the project lock for Treehouse and Herdr ordering, so local homes cannot admit the same last place.
… contribution test
…upervision test expectations
… capacity reservations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge upstream
b062eb94c50a68dd4467fba25346a7d7b4083288into forkhouse, preserving upstream ancestry with true merge commitb11485a69613bf0b42fe1a84d10b4ed3b5f33acb.Upstream had not moved from the scout's pinned tip.
The fork's
mainwas fast-forwarded from47aff866to that pinned tip using a non-force Git push after confirming ancestry.This pull request targets
houseand must land with a merge commit.Upstream commits brought in:
ac0811c4feat(bin): defer spawns beyond a declared per-project capacity, opt-in (feat(bin): defer spawns beyond a declared per-project capacity, opt-in kunchenguid/firstmate#5343)228b27d2fix(bin): name the Lavish read message count by the same label as its section (fix(bin): name the Lavish read message count by the same label as its section kunchenguid/firstmate#6785)8fa25382fix(herdr): make the presentation lock namespace per OS account (fix(bin): make the Herdr presentation lock namespace per OS account kunchenguid/firstmate#6780)0c83b3f4Fix OpenCode arm plugin to decide with the shared supervision predicate (fix(opencode): align supervision arming predicate kunchenguid/firstmate#6809)9b85a955fix(bin): resolve a pending reply only from its own task's status line (fix(bin): resolve a pending reply only from its own task's status line kunchenguid/firstmate#6792)2ce57d00docs(skills): index the six missing agent-only skill triggers (docs(skills): add six missing skills to agent skill trigger index kunchenguid/firstmate#6784)329ad4edtest: make agent process fixtures compatible with multicall sleep (test: make agent process fixtures compatible with multicall sleep kunchenguid/firstmate#6814)3c58ec8efix(bin): keep the supervision host's successor watcher alive after the Stop hook's group is torn down (fix(bin): keep the supervision host's successor watcher alive after the Stop hook's group is torn down kunchenguid/firstmate#6787)0ec1c5a4test: make tmux Claude readiness checks independent of permission footers (test: make tmux Claude readiness checks independent of permission footers kunchenguid/firstmate#6823)b062eb94test(herdr): accept safe exit refusals and clean lab once (test: correct Herdr smoke assertions and cleanup kunchenguid/firstmate#6818)Conflict resolutions:
.agents/skills/operational-home-layout/SKILL.md: retain upstream's capacity entry and house's secondmate harness entry, including its per-mate override.bin/fm-spawn.sh: retain house's bounded project-lock acquisition helper and held-flag ownership; preserve upstream's Orca admission conditions and capacity deferral behavior.bin/fm-wake-lib.sh: add upstream's local-home walker while retaining house's project locks and PID-start ownership checks.Capacity reconciliation:
Admission publishes a process-owned reservation atomically under the shared project-origin lock before either Treehouse acquisition or Herdr presentation ordering can release it.
Other local homes count that pending reservation until metadata from the matching
spawn_gentakes over, without counting the reservation and record twice or letting old PR-ready metadata hide a restarted admission.Success and failed-launch cleanup atomically retire the reservation from the counted namespace before deletion; concurrent readers treat disappearance as retirement while surviving malformed or unreadable reservations still refuse admission.
Counting reaps leases whose original process is proven gone and keeps uncertain ownership occupied.
The local-home walker takes checked registry snapshots and propagates unreadable-registry failures instead of omitting nested holders.
The common path covers all fresh worker backends, including Orca when any project is capped.
Projects with no declared limit retain concurrent spawning.
All ten upstream commits are included;
ac0811c4was reconciled rather than held back.House behavior dropped: none.
Validation:
fm-spawn.shentry point: one scheduled cross-home overlap exhibited oversubscription before reconciliation, with B launching during A's unlocked Treehouse get; the same case defers B with exit 75 after reconciliation, before brief, endpoint, metadata, or backlog changes.bash bin/fm-lint.shand documentation audience checks pass.fm_pending_reply_remote_observationattests/fm-pending-reply.test.sh:1563; the same diagnostic reproduces at line 1530 on the original413014cehouse source snapshot.tests/fm-pending-reply.test.shretains a pre-existing failure:remote_host: unbound variablein the delivery-confirmation fallback regression, atbin/fm-pending-reply-lib.sh:1674here and line 1669 in the original413014cehouse snapshot.Commands:
bash tests/fm-pending-reply.test.shandbash /tmp/fm-hf-firstmate-upstream-sync-1008/house-lint-snapshot/tests/fm-pending-reply.test.sh; the snapshot was exported withgit archive origin/house bin tests.The newly imported token-isolation test passes separately through its unchanged fixture function and production library (
bash /tmp/fm-hf-firstmate-upstream-sync-1008/pending-token-regression.sh).unset TRACEPARENT.readcommand.eb51e07c: update owner comments and operator guidance for matching generations, atomic retirement, and checked registry reads.80fc8fbd: quote the empty reservation-reader assignments; canonical pipeline lint passes.No-mistakes review fixes:
205d3738: remove the duplicate forge-clock initialization approved by firstmate.7d2c1fdc: preserve fresh restart admissions over old PR-ready metadata using the existing generation, make reservation retirement atomic, reject unreadable local registries, and use independent supervision-contract expectations in the OpenCode plugin test.205d3738and pass on7d2c1fdc: restart overlap, retirement during reads, and a nested holder behind an unreadable registry.Test exception:
Firstmate approved the pipeline's inconclusive runtime verdict as an explicit Test exception under the standing targeted-test rule.
The tool reports 5 of 12 scenarios driven; its disposable CLI/report/browser checks use controlled test state and do not establish real-account or full primary-runtime behavior.
The following seven scenarios were not driven live:
Targeted synthetic regressions cover these mechanisms where permitted; full launch, authenticated primary, Herdr, and prompt-guard verification remains incomplete.
This exception is not a clean full-runtime validation pass.
Recorded operator reason:
The capacity and lock-order regressions exercise built CLI behavior with synthetic/offline backend responses and make no real-account launch claim.
The default-on Herdr control smoke exercised the installed Herdr 0.9.1 backend in a named throwaway lab with a stand-in agent, passed, and completed cleanup, despite this task brief lacking its required
--herdr-labdeclaration.Firstmate reviewed this instruction violation in steering message 001, confirmed that the isolated run needs no remediation, and authorized continuing checks and handoff while excluding all further Herdr-driving/default-on Herdr tests.
The smoke is a synthetic built-CLI lab scenario using the installed backend and a stand-in process; it does not verify a real harness or real account.
House CI: House checks passed on pushed head
80fc8fbd33dc30cfe9d119fd58a3ebf02ea13d63.No-mistakes returned
checks-passedwith the explicit Test exception above; the PR is open and not a draft.evidence-artifact: /tmp/fm-hf-firstmate-upstream-sync-1008/evidence.txt
evidence-command: bash /tmp/fm-hf-firstmate-upstream-sync-1008/capture-evidence.sh
evidence-captured: 2026-10-08T15:55:02Z
Pipeline
Updates from git push no-mistakes